State-db I/o metrics #4562
State-db I/o metrics #4562
Conversation
|
It looks like @NikVolf signed our Contributor License Agreement. 👍 Many thanks, Parity Technologies CLA Bot |
marcio-diaz
left a comment
marcio-diaz
left a comment
There was a problem hiding this comment.
Looks good, just a few very little nits. I'll take a deeper look tomorrow.
|
I'd rename |
NikVolf
force-pushed
the
nv-io-stats-sdb
branch
from
37553a0 to
a910533
Compare
NikVolf
force-pushed
the
nv-io-stats-sdb
branch
from
a910533 to
5e90a3b
Compare
Co-Authored-By: Gavin Wood <[email protected]>
| let mut bytes: u64 = 0; | ||
| for (key, (val, rc)) in operation.db_updates.drain() { | ||
| if rc > 0 { | ||
| ops += 1; |
There was a problem hiding this comment.
Field db_updates are calculated trie nodes delta that are going to be written in rocksdb, so I think we should change the name, because it gives the wrong idea that the collected read stats (key value) are comparable with those writes (trie nodes).
We can also get write stats that correspond to our read, that would be field 'storage_updates' and 'child_storage_updates', IIRC those are send to storage cache for update and are complete in term of delta.
But we could also hook some 'write_cache' stat in state-machine overlay-db to compare what are actual post block modif ('storage_updates' fields) against what update the runtime did call. That way, we get an idea of the proportion of updates that are done on a same key. (at
| } else if rc < 0 { | ||
| ops += 1; | ||
| bytes += key.len() as u64; | ||
|
|
There was a problem hiding this comment.
We could store that in a different write counter.
With non archive mode it is actually an additional memory consumption until pruning, but with archive mode it is a no-op.
Or we check mode (it is maybe accessible in client-db).
| let value = self.state.child_storage(storage_key, child_info, &key.1[..])?; | ||
|
|
||
| // just pass it through the usage counter | ||
| let value = self.usage.tally_child_key_read(&key, value, false); |
There was a problem hiding this comment.
So we have something like state_read_cache - state_cache = state-read-db-backend.
One thing that we do not register regarding read query emitted from the runtime will be all scenario such as :
- querying a key that has been change during block processing: in this case the overlay-db from state machine return the value and we do not have stats.
- that's all I can think off :)
We could register them in
substrate/primitives/state-machine/src/ext.rs
Line 157 in 65ad8e9
This allow collecting state cache requests and report cache hit ratio
@arkpar
During the initial sync hit ratio seems to be close to 100%, by the way :)